Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the document to backfill using tx_meta #273

Merged
merged 7 commits into from
Jul 31, 2024

Conversation

amishas157
Copy link
Contributor

@amishas157 amishas157 commented Jul 31, 2024

PR Checklist

PR Structure

  • This PR has reasonably narrow scope (if not, break it down into smaller PRs).
  • This PR avoids mixing refactoring changes with feature changes (split into two PRs
    otherwise).
  • This PR's title starts with the jira ticket associated with the PR.

Thoroughness

  • This PR adds tests for the most critical parts of the new functionality or fixes.
  • I've updated the README with the added features, breaking changes, new instructions on how to use the repository. I updated the description of the fuction with the changes that were made.

Release planning

  • I've decided if this PR requires a new major/minor/patch version accordingly to
    semver, and I've changed the name of the BRANCH to release/_ , feature/_ or patch/* .

What

This PR add instructions on backfill historical data using tx_meta

Why

Airflow runs can be time and money consuming. To ease up the migrations, JS UDF can be convenient.

Known limitations

To be figures: Cost of UDF (they should be cheap most likely)

@amishas157
Copy link
Contributor Author

amishas157 commented Jul 31, 2024

Steps outlined in the document added in this PR. I will add screenshots and links for confirming test results

Library in GCS bucket

Screenshot 2024-07-31 at 2 41 17 PM

https://console.cloud.google.com/storage/browser/_details/stellar-test-js-udf/stellar-base.min.js;tab=live_object?project=crypto-stellar

JS UDF

https://console.cloud.google.com/bigquery?ws=!1m7!1m6!12m5!1m3!1stest-hubble-319619!2sus-central1!3s834c1d13-e58e-40ff-90ec-a7435ec4fe42!2e1
Screenshot 2024-07-31 at 2 44 17 PM

Results confirmations:

Screenshot 2024-07-31 at 2 45 52 PM

Sample check against horizon (testnet) ✅

  • transaction_hash: f5f5b0aaf758896ef8c5b4807f41c77d15c11977eecf2b0e4769d777324a2d11
    js_udf_result: MCBD54KAHHA4AK4DOZWOSX5O5OZ4OI54N24QITDSFLPD7EG2WY2AMAAACYGGCDRL6UBUA
    horizon_result: MCBD54KAHHA4AK4DOZWOSX5O5OZ4OI54N24QITDSFLPD7EG2WY2AMAAACYGGCDRL6UBUA
    Screenshot 2024-07-31 at 2 48 18 PM

Next steps:

  • Discuss and get a review from the team
  • Add js library to final bucket. Bucket to be decied.
  • Add SQL snippet to update column from the results
  • Run the query for historical data

@amishas157 amishas157 marked this pull request as ready for review July 31, 2024 19:51
@amishas157 amishas157 requested a review from a team as a code owner July 31, 2024 19:51
backfill.md Outdated
Comment on lines 49 to 55
)
AS r"""
let txe = StellarBase.xdr.TransactionEnvelope.fromXDR(tx_meta, 'base64');
let tx = txe.feeBump();
let sourceAccount = StellarBase.encodeMuxedAccountToAddress(tx.tx().feeSource());
return sourceAccount

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Shaptic for the helps here 🙌

lint pass?
@amishas157 amishas157 force-pushed the feature/hubble-499/support-backfills-using-txmeta branch from 19d575e to ea28305 Compare July 31, 2024 20:12
backfill.md Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice!

Do you think we should make a new directory for this .md? It seems misplaced in the base directory. Maybe like a notes or scripts directory? Open to naming ideas

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs/? I am also thinking to include migrations directory, where we manually track any udfs run we apply

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup I like that idea

Copy link
Contributor

@chowbao chowbao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

backfill.md Outdated Show resolved Hide resolved
backfill.md Outdated Show resolved Hide resolved
backfill.md Outdated

Following is example to extract FeeAccountMuxed:

```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: add javascript tag here for syntax highlighting

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in d5884a2

backfill.md Outdated Show resolved Hide resolved
backfill.md Outdated
Comment on lines 52 to 55
let txe = StellarBase.xdr.TransactionEnvelope.fromXDR(tx_meta, 'base64');
let tx = txe.feeBump();
let sourceAccount = StellarBase.encodeMuxedAccountToAddress(tx.tx().feeSource());
return sourceAccount
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let txe = StellarBase.xdr.TransactionEnvelope.fromXDR(tx_meta, 'base64');
let tx = txe.feeBump();
let sourceAccount = StellarBase.encodeMuxedAccountToAddress(tx.tx().feeSource());
return sourceAccount
return StellarBase.encodeMuxedAccountToAddress(
StellarBase.xdr.TransactionEnvelope.fromXDR(tx_meta, 'base64')
.feeBump()
.tx()
.feeSource()
);

😉

backfill.md Outdated Show resolved Hide resolved
@amishas157 amishas157 merged commit ffb4b4a into master Jul 31, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants